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ABSTRACT 


The objective of this summer’s work was to attempt to enhance TAG’S ability to 
measure the outcomes of its efforts to transfer NASA technology. By reviewing existing 
literature, by explaining the economic principles involved in evaluating the economic impact 
or technology transfer, and by investigating the LaRC processes our William & Mary team 
has been able to lead this important discussion. 

In reviewing the existing literature, we identified many of the metrics that are currently 
being used in the area of technology transfer. Learning about the LaRC technology transfer 
processes and the metrics currently used to track the transfer process enabled us to compare 
other R&D facilities to LaRC. 

In the working paper “Measuring the Economic Impact of Technology Transfer From a 
National Laboratory: A Primer,” we discuss and diagram impacts of technology transfer in 
the short run and the long run. Significantly, this paper serves as the basis for analysis and 
provides guidance in thinking about what the measurement objectives ought to be. 

By focusing on the SBIR Program, valuable information regarding the strengths and 
weaknesses of this LaRC program are to be gained. A survey was developed to ask 
probing questions regarding SBIR contractors’ experience with the program. Specifically 

are interested in finding out whether the SBIR Program is accomplishing its mission, if 
the SBIR companies are providing the needed innovations specified by NASA and to what 
extent those innovations have led to commercial success. 

We also developed a survey to ask COTRs, who are NASA employees acting as 
technical advisors to the SBIR contractors, the same type of questions, evaluating the 
successes and problems with the SBIR Program as they see it. This survey was developed 
to be implemented interactively on computer. 

It is our hope that the statistical and econometric studies that can be done on the data 
collected from all of these sources will provide insight regarding the direction to take in 
developing systematic evaluations of programs like the SBIR Program so that they can 
reach their maximum effectiveness. 



INTRODUCTION 


The Technology Applications Group (TAG) is fairly new to LaRC, having only been 
formed within the last two years. Our work this summer was geared toward enhancing 
TAG’s ability to measure the outcomes of its efforts to transfer LaRC technology. Our 
work began last semester as we reviewed the TAG mission statements and organizational 
charts. During our first weeks at NASA, we extensively reviewed the existing literature 
pertaining to the measurement of technology transfer and its impact on the economy and 
throughout the summer we continuously attempted to broadened our understanding of the 
TAG transfer process by observing and talking with the individuals involved in the transfer 
activities. 

Our paper “Measuring the Economic Impact of Technology Transfer From a National 
Laboratory: A Primer” laid the ground work for guiding our discussions regarding 
technology transfer measures. The paper provides guidance in thinking about measurement 
objectives and the associated metrics to use when attempting to measure the impact of 
technology transfer on the economy, and it discusses many of the limitations and 
difficulties in the measurement process. 

Combining the economic theory with what we learned from our research and 
observations, we outlined a potential list of metrics that arc specific to TAG at LaRC. This 
list included input, output and outcome measures and does not speak to the measurement 
process, which is complex and dynamic as it requires the cooperation and coordination of 
many people at all levels in an organization. 

As an example of performance evaluation, we turned our focus toward a project to 
investigate a specific program. The Small Business Innovation Research Program (SBIR) 
was chosen. In this effort, we wrote, designed, and developed two surveys this summer 
to gather data about the participants of the SBIR Program. By statistically analyzing the 
data that we collect, it is our hope to provide information that will maximize the 
effectiveness of the SBIR Program as well as show by example the advantages that 
organizations can have in decision making when the relevant metrics are analyzed. 

In each of the following sections of this paper we briefly discuss each element of our 
work here at NASA this summer. 


SECTION I 

BRIEF DESCRIPTION OF SOME OF THE DISCOVERIES MADE IN REVIEWING 
THE LITERATURE 

In reviewing the existing literature, we identified many of the metrics that are 
currendy being used in the area of technology transfer. Researchers agree that basing 
current policy on past studies is difficult because the average time lag between the 
discovery, development , production and marketing of an innovation is 15-20 years. They 
also agree that advances in technology have a positive effect on the economic growth of a 
nation, but they disagree on how large the effect is. 

Researchers agree that there are different facets of R&D that overlap. Applied science 
research leads to more commercial production, pure science research may often lead to 
“break through discoveries” that may take a long time to develop into commercial products. 
“Push technologies” are usually less marketable than “pull technologies.” Pull technologies 
refer to innovations that have been discovered as a result of consumer or producer demand, 
push technologies refer to innovations that have not been fully developed and lack 
production and/or marketing momentum. 



SECTION II 


INVESTIGATION OF THE LaRC TECHNOLOGY TRANSFER PROCESS 

The technology available to TAG for transfer requires a “push” strategy and 
therefore the objectives and relevant metrics must be designed to capture TAG’S 
effectiveness in assisting in the development of innovations into marketable products. The 
diagram below summarizes the TAG transfer process. At each stage of this diagram, 
metrics can be developed to determine whether specific objectives are being met 
effectively. 


LaRC TECHNOLOGY TRANSFER PROCESS DIAGRAM 



By tracking the activities in the transfer process and analyzing the data continuously 
over time, the most successful strategies can be identified and used to improve the overall 
effectiveness of TAG. 






SECTION III 


METRICS DISCUSSION 

The best metrics are associated with clearly stated objectives. If a program or a job 
description is not precise in it’s expectation of what is to be accomplished, the relevant 
metrics cannot be adequately identified. Clear objectives are a most important element of 
evaluating success. Equally important are consistent, quantifiable measures that are 
systematically collected , analyzed and that the resulting information is used in determining 
future decisions. 

METRICS THAT MAY BE USEFUL TO NASA LaRC 

• Input Metrics 

— Number of spin-offs and spin-ons 

— Number of patents 

— Number of TOPS exhibits (paten table/non-patentable, patented/not patented, 
licensed/not licensed) 

— Number of COOPPRs 

— Number of contacts (post technology transfer) 

— Number of contacts (pre technology transfer) 

— Time spent on technology transfer 
— Use of facilities 

• Output Metrics 

Number of MOAs/MOUs and number of MOAs/MOUs leading to 
commercialization 

Number of university/industry partnerships 
Technical briefs/papers (published, requested, presented) 

Technical problems solved 

Number and dollar value of licenses and sublicenscs 
Number of software packages released 

• Outcome Metrics 

Number and dollar value of commercialization from TOPS, COOPPRs, 

Cooperative Agreements, MOAs, MOUs, SAAs and SBIRs 

Commercial sales 

Royalties 

Cost savings 

Productivity gains 

User satisfaction 

Short-run economic impact on income and employment 

This was a tentative grouping of metrics that were presented during the 10 
week summer session. Thoughts on these and other metrics are not complete. The 
important discussion should include metrics like these, but as mentioned earlier, metrics are 
only part of the process. 
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SECTION IV - THE PRIMER 


Having identified the need for a discussion of the economic implications of the 
discussions that were taking place at NASA about the impact that technology has on the 
economy, a primer was written. My contribution to this effort was in creating the graphical 
representations of the concepts discussed in the paper. A full copy of the paper is enclosed 
along with this final report. The executive summary follows. 

MEASURING THE ECONOMIC BENEFITS OF TECHNOLOGY TRANSFER 
FROM A NATIONAL LABORATORY: A PRIMER 

Robert Archibald, Professor of Economics 
David Finifter, Professor of Economics 
and Director of the Thomas Jefferson Program in Public Policy 
Nanette Smith, Graduate Student 
Thomas Jefferson Program in Public Policy 

- Executive Summary - 

One measures the economic benefits of a particular event by comparing the 
state of the economy after the event has happened to the state of the economy 
assuming the event had not happened. Given this understanding of 
economic benefits, in this paper we reach the following conclusions about the 
economic benefits of technology transfer from national labs. 

• In the long run, successful technology transfer increases the 
productivity of the country. As a result the per capita income 
generated by the U. S. economy will rise. Technology transfer from 
national labs will spur economic growth. 

• In the long run, the number of jobs in the economy is unaffected. 
The characteristics of the jobs are different, however. On average, 
the real compensation of jobs will increase, and therefore 
technology transfer does create higher paying (better) jobs. 

• In the long run, the economic competitiveness of U.S. firms will be 
enhanced, i.e., the balance of trade will be improved. 

• In the short run, outcomes are less clear. The results of increases in 
technology transfer activity depend upon the stage of the business 
cycle and in any case are only temporary. 

These conclusions have clear implications for the measurement efforts being 
undertaken at NASA labs. To correctly measure the benefits of technology 
transfer, NASA labs need to focus on the long run consequences of their 
activities. 
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SECTION V 


SURVEYS 

By studying the SBIR Program information might be provided that could help to 
maximize its effectiveness. This study can also be useful in demonstrating by example the 
advantages that organizations can have in decision making when the relevant metrics are 
analyzed. The SBIR Program awards contracts to small businesses who present a viable 
case for doing research that NASA solicits. These contracts are awarded in two phases. A 
Phase I contract is awarded to begin the research, a Phase II contract is awarded after the 
Phase I has been completed , it is the second step in the process and may result in a product 
or process that can not only be used by NASA, but also may become commercially 
successful. If a company is awarded a Phase I or Phase II contract, a NASA employee is 
assigned to the project as a COTR to provide technical assistance. 

Our team developed and sent a survey to the companies who participated in the 382 
SBIR contracts awarded form 1985 to present. The objective of the survey is to collect data 
from the SBIR firms regarding the success of their innovation that can be analyzed to gain 
insight in answering the following questions. Did their solicitation result in an innovation 
that was or could be used by NASA? Did their innovation result in a commercially viable 
product? Did the SBIR award impact the economy? 

Since COTRs can also provide valuable information about the SBIR Program, we 
developed a second survey to be completed by COTRs. This survey was developed to be 
implemented interactively on computer. 

Enclosed are copies of the two surveys and a print out of the automated survey input 
screens. 


Ideas Generated From the Study: 

Economic Impact of Technology Transfer 

Idea 1 

Counting the number of jobs that are generated by technology transfer is a political 
argument for justifying the continued funding of federal labs, but it is a short run and 
incomplete argument that does not reflect the long run benefits of increasing the nation’s 
global competitiveness, gross domestic product per capita, and quality of life as Congress 
determines how it will allocate shrinking federal resources. See “Measuring The Economic 
Impact of Technology Transfer From a National Laboratory: A Primer” which was 
developed by the William & Mary team this summer to explain the economic theory behind 
this claim. 

Idea 2 

By changing the contracting process to require that contractors provide NASA with the 
information needed to better estimate the potential economic benefit of the transfer, 
communities could be better served. As firms forecast potential profit and report on actual 
profit that results from NASA research, the NASA contribution becomes more tangible and 
measurable. The information needed should include data that will indicate the potential 
savings and/or profits that will and do result from the transferred technology. Other 
questions to investigate are: Who benefits beyond the company receiving the innovation , 
what is the benefit, and how much is it? How long does/did each step in the process take? 



How do the benefits/costs vary over time? Are there losers, if so who are they and how 
much do they lose? To capture the economic impact of technology transfer it will require 
a long term informational commitment between NASA and the contracting firm. How can 
NASA staff better maintain this information link? 

Toward TAG Metrics 

Idea 1 

Assigning personnel to analyze the metrics being captured and to develop the evolution of 
the performance measurement process could enhance TAG’S ability to quantify the success 
of its technology transfer efforts. The analyst(s) should be able to link metrics to 
organizational objectives in order to evaluate the strengths and weaknesses of the 
technology transfer processes. The analyst(s) should be able to identify additional 
information requirements and be able to recommend and implement adaptations to the 
measurement process. By being able to discern process patterns allows for discoveries to 
be made that could lead to increased transfer activity. 

Idea 2 

When a NASA technology is being considered for licensing by more than one firm could a 
bidding process increase the probability of its commercial success? By opening the 
licensing for bidding, increased commitment toward commercialization may occur as an 
outcome of the higher cost to the firm that wins the bid. The bidding process also permits 
NASA to get a feel for the market value of the innovation being transferred, thus gaining 
insightful information that could be used in future technology transfer efforts. 


SBIR/STTR Programs 
Idea 1 

Could the SBIR/STTR Programs be improved to enhance commercialization of 
technology? An analysis of the SBIR/STTR Programs might help to determine whether 
solicitation topics written for SBIR/STTR contracts can or should be focused more toward 
“applied” R&D, which may result in increased commercial activity. 

Idea 2 

Investigating the possibility of incorporating problem statements into SBIR/STTR 
program might lead to increased commercial activity as R&D efforts are focused on 
process or product improvement. This might be accomplished by matching SBIR/STTR 
firms with problem statement firms through personal contact or through solicitation write- 
ups. 



